Add ability to process local docs #318
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ppinchuk
added
enhancement
Contributor
There was a problem hiding this comment.
Pull Request Overview
This PR adds functionality to process local documents in addition to web-based URLs. The implementation introduces a new load_local_docs utility and corresponding file loader services for HTML and PDF files, enabling users to provide local file paths alongside URL-based sources.
Key Changes
- Added
AsyncLocalFileLoadersupport for processing local PDF and HTML files - Introduced
known_local_docsparameter to allow users to specify local document paths - Modified processing pipeline to check local documents before URL-based searches
- Updated class references from
AsyncFileLoadertoAsyncWebFileLoaderfor clarity
Reviewed Changes
Copilot reviewed 15 out of 16 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| compass/utilities/io.py | New utility module for loading local documents |
| compass/services/cpu.py | Added functions to read local PDF files (with and without OCR) |
| compass/services/threaded.py | Added HTMLFileLoader service and functions to read local HTML files |
| compass/scripts/process.py | Updated main processing function to support local documents and modified processing order |
| compass/scripts/download.py | Added load_known_docs function for loading local documents |
| compass/utilities/nt.py | Added known_local_docs field to ProcessKwargs namedtuple |
| compass/utilities/init.py | Exported new load_local_docs function |
| compass/validation/location.py | Updated class reference from AsyncFileLoader to AsyncWebFileLoader |
| compass/validation/content.py | Made legal_text_validator parameter optional in parse_by_chunks |
| compass/extraction/apply.py | Added logic to skip legal text and date validation for known documents |
| compass/web/website_crawl.py | Updated class references and fixed documentation typo |
| tests/python/unit/utilities/test_utilities_io.py | New test file for local document loading functionality |
| tests/python/unit/utilities/test_utilities_base.py | Minor docstring correction |
| tests/python/unit/validation/test_validation_location.py | Added missing @pytest.mark.asyncio decorator |
| tests/python/integration/test_integrated.py | Updated class references from AsyncFileLoader to AsyncWebFileLoader |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Add option to process local documents, similar to how we allow processing known URLs